COMMENT ⊗   VALID 00002 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.S LISP Timing Evaluation Project
C00010 ENDMK
C⊗;
.S LISP Timing Evaluation Project

//pers John#McCarthy, Richard#Gabriel sra ---



The Lisp Timing Evaluation Project began at the end of February and
is slowly taking shape. Originally simply a timing project, other
evaluations are being considered. 

The thrust of the project is to provide both objective and subjective
bases for rational choice among the various Lisps and Lisp systems
available today or to be available in the near future. The objective
measures are to be provided through the use of benchmarks which will
be run on the various systems in the test with measurements made
in terms of CPU time. These benchmarks will be/are being provided
by people at the various sites in order to provide a range of
interesting benchmarks, not merely a few artificial ones. 
This latter point is important for several reasons. First,
it will test the various systems with the range of real applications
that are going on. Second, it will bring too light the esoteric
features used by research programmers (and, hence, shed some
light on Hearn's claim that a relatively small subset of Lisp 
will suffice for most research programming in Lisp).

In the event that benchmarks covering the range of desired 
features are not available, `artificial' benchmarks will be
supplied. Either benchmarks will be solicited from people and systems
that are known to perform such operations or they will be 
written explicitly for the situation.

The subjective measures are descriptions of the systems provided
by the volunteers along with experiences with the translating the
various benchmarks: since the benchmarks are not restricted in any
way a translation phase for each site is required. The tools and
problems associated with each translation (with the original and
translated programs as evidence) can be interpreted as a measure
of the expressive power of a given Lisp/Lisp system.

A final measure of the non-language efficiency will be attempted,
though there will be some technical problems here. For example,
garbage collection, system paging overhead, and external loads
on the machines affect the `usability' of a Lisp without being
directly associated with the language.

The following is a list of the systems to be tested as known at this point:

.begin verbatim

	InterLisp on MAX, Dolphin, Dorado
	MacLisp on SAIL
	NIL on S-1
	InterLisp on SUMEX
	UCILisp on Rutgers
	SpiceLisp on PERQ
	Lisp Machine (Symbolics, CADR)
	MacLisp on AI, MC, NIL on VAX
	InterLisp on F2
	Standard Lisp on TOPS-10, B-1700 
	Lisp370
	TLC-Lisp on Z-80
	muLisp on Z-80
	Muddle on DMS
	Rutgers Lisp
	Multics MacLisp
	Jericho InterLisp
	Cromemco Lisp on Z80
	Franz Lisp on VAX UNIX
	UTILisp

.end

Some proposed features of Lisps that will be tested, either
by naturally occurring benchmarks or by artificial ones are:

.begin verbatim

	Array reference and storage (random access)
	Array reference and storage (matrix access)
	Array reference and storage (matrix inversion)
	Short list structure (records, hunks...)
	Long list structure (cdr access)
	CAR heavy structures
	CDR heavy structures
	Interpreted function calls
	Compiled function calls
	Smashed function calls
	Table function calls (FUNCALL, SUBRCALL)
	Tail recursion (?)
	Block compiling
	Reader speed
	Property list structures
	Atom structures (saturated obarrays)
	Internal loops
	Trigonometric functions
	Arithmetic (floating and fixed)
	Special variable lookup
	Local variable lookup
	CONS time
	GC time
	Compiled code load time
	EQ vs EQUAL test time
	Arithmetic predicates
	Type determination

.end

The remaining work is to complete the gathering and initial
evaluation of the benchmarks, collecting results from the various
systems, collection of system descriptions, factoring out Lisp
versus system overhead from raw data, evaluation of the impact
of system overhead and garbage collection, and publication and
distribution of results. Since some machines and systems are
not yet available for testing, the project will be completed
by June 1982.